Embodiments of the present invention relate to parallel execution of machine learning programs, and more specifically to hybrid parallelization strategies for machine learning programs on top of MapReduce.